Summary

zipsync is a tool to pack and unpack zip archives. It is designed as a single-purpose tool to pack and unpack build cache entries.

Details

Unpack

load archive -> parse central dir -> read metadata
scan filesystem & delete extraneous entries
for each entry (except metadata):
  if unchanged (sha1 matches) => skip
  else extract (decompress if needed)

Pack

for each file F
  write LocalFileHeader(F)
  stream chunks:
    read -> hash + crc + maybe compress -> write
  finalize compressor
  write DataDescriptor(F)
add metadata entry (same pattern)
write central directory records

Supported compression types are store (no compression), deflate (level 9), auto (switches between store/deflate based on file extension).

Constraints

Though archives created by zipsync can be used by other zip compatible programs, the opposite is not the case. zipsync only implements a subset of zip features to achieve greater performance.

What's wrong with the current setup?

The current setup cleans target directories when unpacking; then the build cache entry is unpacked. This setup ends up deleting and rewriting a lot of the same files.

Pros

With tar + gzip files are archived first and compressed second. This allows the compression to work across file boundaries. Duplicate content across files can be efficiently compressed.

Cons

Since compression is the last step, uncompressing the archive is required to inspect it. To enumerate the archive, it must be uncompressed first.
It does not clean the target directory so a rm -rf step is required.

Requirements

zipsync was created with the following constraints in mind

Optimize for partial unpack scenario

Optimize for unpack performance. Most of the build cached files already exist on disk and there is a good chance for them to be already in the expected state.

Only write files when needed

This will minimize the number of write syscalls. Also, if the kernel has already cached the file from a recent read, the cache remains intact if we don't needlessly delete and rewrite the file.

Clean extra files and directories

This will remove the need to run rm -rf on the target directories. More time saved

Disallow symlinks

Symlinks in build cache entries are not supported. This will remove the need to scan the target directories for symlinks before running tar.

Why zip

zip was picked because:

• It is a well understood format. This will keep malware scanning happy.
• Easy to inspect build cache entries. The built-in os zip tools can be used to browse or extract files.
• The files are compressed first then added to the archive. This allows us to inspect the archive contents without paying the cost to decompress its entire contents allowing efficient unpacking.

How it was tested

• zipsync tests
• rebuild then build with node apps/rush/lib/start-dev.js --debug build --verbose -t module-minifier

==[ @rushstack/heft-typescript-plugin (build) ]==================[ 19 of 25 ]==

Build cache hit.
Cache key: rushstack+heft-typescript-plugin-_phase_build-1ce783d05b5d5382cf8f9652ff0f14416482830c
Using zipsync to restore cached folders.
Restored 55 files from cache.
Skipped 55 files that were already up to date.
Successfully restored output from the build cache.
Invoking: heft run --only build -- --clean
 ---- build started ----
[build:clean] Deleted 0 files and 3 folders
[build:typescript] Using TypeScript version 5.8.2
[build:typescript] Copied 2 files and linked 0 files
[build:lint] Using ESLint version 9.25.1
[build:api-extractor] Using API Extractor version 7.52.9
[build:api-extractor] Analysis will use the bundled TypeScript version 5.8.2
 ---- build finished (3.138s) ----
-------------------- Finished (3.142s) --------------------
pnpm-sync: Starting operation for /Users/bharatmiddha/code/rushstack-zipsync/heft-plugins/heft-jest-plugin/node_modules/.pnpm-sync.json
pnpm-sync: Synced 54 files in 8 ms
"@rushstack/heft-typescript-plugin (build)" was restored from the build cache.

Benchmark Results

This document contains performance measurements for packing and unpacking a synthetic dataset using tar, zip, and zipsync.

The dataset consists of two directory trees (subdir1, subdir2) populated with 1000 text files each.

zipsync scenarios

• "all-existing": unpack directory is fully populated with existing files
• "none-existing": unpack directory is empty
• "partial-existing": unpack directory contains half of the files

zip and tar scenarios clean the unpack directory before unpacking. This time is included in the measurements because
zipsync internally handles cleaning as part of its operation.

System

Iterations: 100

Compressed (baseline: tar-gz)

Unpack Phase

Pack Phase

Uncompressed (baseline: tar)

Unpack Phase

Pack Phase